Obstacle avoidance method and device and movable platform

ABSTRACT

An obstacle avoidance method includes obtaining a depth image photographed by a photographing device mounted at a movable platform, recognizing a moving object based on the depth image, determining a moving speed vector of the moving object, determining a potential collision region where the moving object is likely to collide with the movable platform based on the moving speed vector of the moving object; and controlling the movable platform to perform an obstacle avoidance process in the potential collision region.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of International Application No.PCT/CN2017/120249, filed on Dec. 29, 2017, the entire content of whichis incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to the field of unmanned aerial vehicle(UAV) technology and, more particularly, to an obstacle avoidancemethod, an obstacle avoidance device, and a movable platform.

BACKGROUND

With the increasing popularity of UAVs, more people are using UAVs foraerial photographing. However, it is a challenge for a user who neverused a UAV before to operate the UAV. A minor operation mistake maycause a crashing accident. Thus, the user needs assistance in operatingthe UAV to avoid obstacles.

SUMMARY

In accordance with the disclosure, there is provided an obstacleavoidance method including obtaining a depth image photographed by aphotographing device mounted at a movable platform, recognizing a movingobject based on the depth image, determining a moving speed vector ofthe moving object, determining a potential collision region where themoving object is likely to collide with the movable platform based onthe moving speed vector of the moving object; and controlling themovable platform to perform an obstacle avoidance process in thepotential collision region.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of an obstacle avoidance method according to anexample embodiment of the present disclosure.

FIG. 2 is a flowchart of a method for obtaining an optical flow vectorof a target feature point according to an example embodiment of thepresent disclosure.

FIG. 3 is a flowchart of a method for recognizing a moving objectaccording to an example embodiment of the present disclosure.

FIG. 4A is a schematic diagram illustrating optical flow vectors oftarget feature points according to an example embodiment of the presentdisclosure.

FIG. 4B is a schematic diagram illustrating clustering of the opticalflow vectors in FIG. 4A.

FIG. 5 is a structural block diagram of an obstacle avoidance deviceaccording to an example embodiment of the present disclosure.

FIG. 6 is a structural block diagram of another obstacle avoidancedevice according to an example embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Technical solutions of the present disclosure will be described withreference to the drawings. It will be appreciated that the describedembodiments are some rather than all of the embodiments of the presentdisclosure. Other embodiments conceived by those having ordinary skillsin the art on the basis of the described embodiments without inventiveefforts should fall within the scope of the present disclosure.

When a first component is referred to as being “fixed” to a secondcomponent, the first component may be directly fixed to the secondcomponent or may have a third component between the first component andthe second component. When the first component is referred to as being“connected” to the second component, the first component may bedirectedly connected to the second component or may have the thirdcomponent between the first component and the second component. Unlessotherwise defined, all technical and scientific terms used herein havethe same meaning as commonly understood by those of ordinary skill inthe art to the disclosure pertains. The terms used in the description ofthe present disclosure are for the purpose of describing particularembodiments of the present disclosure and are not intended to limit thepresent disclosure. The term “and/or” used herein includes any and allcombinations of one or more of the associated listed items.

Technical solutions of the present disclosure will be described withreference to the drawings. The features described in the embodiments ofthe present disclosure may be combined with each other under ano-conflict condition.

In the existing technology, an obstacle avoidance process is oftenperformed on movable platforms such as automobiles and unmanned aerialvehicles (UAV) in the following manners.

In one example, ultrasonic and infrared raging modules may be used. Inthis case, the operation is simple. When an object is detected, abraking action is performed. The braking action causes substantialnoises to the detection, which in turn cause inadvertent brakingactions. Further, the ranging only works for a short distance, and iscompletely useless when the movable platform is moving at a high speed.

In another example, a visual camera captures images and a machine visionalgorithm is used to calculate depth images based on the capturedimages. For example, an RGB-D or a Time-of-Flight (ToF) module may beused to obtain the depth images. The obstacle avoidance process isperformed based on the depth images. However, the depth images arecalculated with a no-motion assumption, that is, it is assumed thatobjects in the captured images do not move relative to the Earth. Forthe movable platforms (e.g., UAVs) flying at a high altitude, theassumption generally holds true. However, for the moveable platformshovering above the ground, the assumption generally does not hold true.It will be ineffective in avoiding automobiles or pedestrians.

Self-driving moveable platforms (e.g., self-driving cars) often performthe obstacle avoidance process based on Light Detection and Ranging(LiDAR) devices. However, LiDAR devices are costly and bulky.

To solve the above technical problems, the embodiments of the presentdisclosure provide an obstacle avoidance method. A photographing devicemounted at a movable platform obtains depth images. Based on the depthimages, a moving object is recognized and a moving speed vector of themoving object is determined. Based on the moving speed vector of themoving object, a region where the moving object may collide with themovable platform is determined, such that the movable platform iscontrolled to perform the obstacle avoidance process in the region ofpotential collision. In the present disclosure, the depth images areused to recognize the moving object. The movable platform is controlledto avoid the moving object, thereby improving the safety and userexperience of the movable platform. The obstacle avoidance methodaccording to the present disclosure is particularly effective inavoiding the moving objects, when the movable platform is a car movingon the ground or other movable object, or a UAV hovering above theground.

Because the embodiments of the present disclosure involve depth imageprocessing, for the convenience of understanding the technical solutionof the present disclosure, certain parameters involved in thephotographing device are described below. Taking a camera as an example,the camera model can be described using the following equation:

${\begin{bmatrix}u \\v \\1\end{bmatrix} = {{K\left\lbrack R \middle| T \right\rbrack}\begin{bmatrix}x_{w} \\y_{w} \\z_{w} \\1\end{bmatrix}}},$

where R is a rotation matrix of the camera, [u v 1]^(T) representstwo-dimensional (2D) points in pixel coordinates, [x_(w) y_(w) z_(w)1]^(T) Represents Three-Dimensional (3D) Points in World coordinates,and matrix K is a camera calibration matrix (i.e., intrinsic parametersof the camera). For a finite projective camera, the matrix K includesfive intrinsic parameters:

${K = \begin{bmatrix}\alpha_{x} & \gamma & \mu_{0} \\0 & \alpha_{y} & v_{0} \\0 & 0 & 1\end{bmatrix}},$

where a_(x)=fm_(x), a_(y)=fm_(y), f is a focal length, m_(x) and m_(y)are scale factors (i.e., numbers of pixels in a unit distance) in x andy directions, respectively, γ is a skew parameter between x axis and yaxis as pixels in a CCD camera are not square, and μ₀ and v₀ areprincipal points.

FIG. 1 is a flowchart of an obstacle avoidance method according to anexample embodiment of the present disclosure. The method may beperformed by an obstacle avoidance device. The obstacle avoidance devicemay be mounted at a movable platform or a ground terminal. As shown inFIG. 1, the method includes the following.

At 101, a depth image photographed by a photographing device mounted ata movable platform is obtained.

In some embodiments, the movable platform includes, but is not limitedto, any one of an unmanned aerial vehicle, an automobile, VR glasses,and AR glasses.

In some embodiments, the photographing device includes at least onecamera.

In some embodiments, two cameras separated by a pre-set distancesimultaneously capture two images of a same scene. A 3D matchingalgorithm is used to process the two images to obtain the depth image.The example is intended to be illustrative and does not limit thepresent disclosure. The method for obtaining the depth image may includeany one of methods in the existing technology and is not limited by thepresent disclosure.

At 102, a moving object is recognized based on the depth image.

In some embodiments, an optical flow vector of a target feature point inthe depth image is obtained. The moving object is recognized based onthe optical flow vector of the target feature point.

In some embodiments, the method for obtaining the optical flow vector ofthe target feature point includes at the least the following.

In one embodiment, because most objects in the photographed view arestationary and only a few objects are moving, the feature points on thestationary objects are the majority of all feature points on thephotographed depth image while the feature points of the moving objectsare the minority of all feature points on the photographed depth image.Directions of the optical flow vectors of the feature points of themoving objects are different from directions of the optical flow vectorsof the feature points of the stationary objects. Thus, the optical flowvectors of the feature points of the moving objects are selected as theoptical flow vectors of the target feature points by clustering theoptical flow vectors of the feature points in the depth image.

In another embodiment, the optical flow vectors of the feature points ofany objects other than the stationary objects (i.e., optical flowvectors of the target feature points) may be selected from the depthimage photographed by the photographing device mounted at the movableplatform through a visual odometry (VO) algorithm carried by the movableplatform. FIG. 2 is a flowchart of a method for obtaining an opticalflow vector of a target feature point according to an example embodimentof the present disclosure. As shown in FIG. 2, the method includes thefollowing.

At 1011, feature points are extracted from the depth image photographedby the photographing device mounted at the movable platform according toa pre-set corner detection algorithm.

In some embodiments, to reduce the amount of calculation, a method forgenerating a sparse matrix is adopted to extract the feature points fromthe depth image photographed by the photographing device. Corners areselected from the depth image as the feature points without losinggenerality. The corner detection algorithm may include, but is notlimited to, a features from accelerated segment test (FAST) algorithm, asmallest univalue segment assimilating nucleus (SUSAN) algorithm, or aHarris corner detection algorithm.

In one embodiment, the Harris corner detection algorithm is adopted asan example, and matrix A is defined as a structure tensor of the depthimage:

${A = {{\Sigma_{u}\Sigma_{v}{{w\left( {u,v} \right)}\begin{bmatrix}I_{x}^{2} & {I_{x}I_{y}} \\{I_{x}I_{y}} & I_{y}^{2}\end{bmatrix}}} = \begin{bmatrix}{\langle I_{x}^{2}\rangle} & {\langle{I_{x}I_{y}}\rangle} \\{\langle{I_{x}I_{y}}\rangle} & {\langle I_{y}^{2}\rangle}\end{bmatrix}}},$

where I_(x) and I_(y) are gradient information of a point of the depthimage in x and y directions, respectively. Based on the matrix A, acorner point response value (M_(c)) of the point of the depth image isobtained:

M _(c)=λ₁λ₂ −k(λ₁+λ₂)²=det(A)−ktrace²(A)

where det(A) is the determinant of the matrix A, trace (A) is the traceof the matrix A, and k is the tunable sensitivity parameter. A thresholdM_(th) can be set, and when M_(c)>M_(th), the corner point is determinedto be a feature point.

At 1012, optical flow vectors of the feature points are determinedthrough tracking relative positions of the feature points in two imageframes.

In some embodiments, the relative positions of the obtained featurepoint in two or more image frames are tracked. Based on displacements ofthe relative positions of the feature point in two or more image frames,the optical flow vector of the feature point is determined. Taking twoimage frames as an example, after tracking the relative positions of theobtained feature point in the two image frames, the displacement h ofthe relative positions of the feature point in the two image frames isiteratively obtained according to the following equation:

$\left\{ \begin{matrix}{h_{0} = 0} \\{h_{k + 1} = {h_{k} + \frac{\Sigma_{x}{w(x)}{{F^{\prime}\left( {x + h_{k}} \right)}\left\lbrack {{G(x)} - {F\left( {x + h_{k}} \right)}} \right\rbrack}}{\Sigma_{x}{w(x)}{F^{\prime}\left( {x + h_{k}} \right)}^{2}}}}\end{matrix} \right..$

For each feature point, the displacement is detected twice. In onedetection, a succeeding image frame is set as F(x) and a preceding imageframe is set as G(x). Based on the above equation, a first displacementh of the relative position of the feature point on the succeeding imageframe relative to the relative position of the feature point on thepreceding image frame is obtained. In another detection, the precedingimage frame is set as F(x) and the succeeding image frame is set asG(x). A second displacement h′ of the relative position of the featurepoint on the preceding image frame relative to the relative position ofthe feature point on the succeeding image frame is obtained. If arelationship between the first displacement and the second displacementsatisfies a first priori condition, the optical flow vector of thefeature point is determined to be h. For example, the first prioricondition may be, but is not limited to, h=−h′. In some embodiments, thefirst priori condition may also be h=−h′+a, where a is a pre-set errorand is a constant.

The above embodiments are merely illustrative and not limiting thepresent disclosure. In practical applications, the displacement for eachfeature point may be determined only once. For the determination methodthereof, reference can be made to the above embodiments and the detailswill not be repeated here.

At 1013, based on three-dimensional coordinates of the feature points ina world coordinate system in the two image frames, optical flow vectorsof the feature points of stationary objects are removed from theobtained optical flow vectors of the feature points to obtain theoptical flow vectors of the target feature points. That is, theremaining ones of the optical flow vectors are taken as the optical flowvectors of the target feature points.

In one embodiment, based on the three-dimensional coordinates of thefeature point in the world coordinate system in the two image frames,the position and attitude of the photographing device when photographingthe two image frames are determined. Based on the position and theattitude of the photographing device when photographing the two imageframes, a random sampling consensus (RANSAC) algorithm is used to obtainthe optical flow vectors of the feature points other than the featurepoints of the stationary objects. The algorithm can be described in thefollowing equation:

${{sp_{c}} = {{K\left\lbrack R \middle| T \right\rbrack}p_{w}}},{{s\begin{bmatrix}u \\v \\1\end{bmatrix}} = {{\begin{bmatrix}f_{x} & \gamma & u_{0} \\0 & f_{y} & v_{0} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}r_{11} & r_{12} & r_{13} & t_{1} \\r_{21} & r_{22} & r_{23} & t_{2} \\r_{31} & r_{32} & r_{33} & t_{3}\end{bmatrix}}\begin{bmatrix}x \\y \\z \\1\end{bmatrix}}},$

where R is a rotation matrix having the position and the attitude of thephotographing device when photographing the image frames as the priori.p_(c) is the two-dimensional coordinates of the feature point in thedepth image. p_(w) is the three-dimensional coordinates of the featurepoint in the world coordinate system.

In another embodiment, based on a pre-set second priori condition, anessential matrix corresponding to the three-dimensional coordinates ofthe feature points in the world coordinate system in the two imageframes is determined. Based on the calculated essential matrix, theRANSAC algorithm is used to obtain the optical flow vectors of thefeature points in the depth image other than the feature points of thestationary objects.

For example, assuming that the three-dimensional coordinates of thefeature point in the world coordinate system in the two image frames arey and y′, respectively, the essential matrix E is determined based onthe second priori condition by the following equation:

(y′)^(T) Ey=0

Further, the RANSAC algorithm is used to obtain the optical flow vectorsof the feature points in the depth image other than the feature pointsof the stationary objects, that is, the optical flow vectors of thetarget feature points. The operation principle and execution process ofthe RANSAC algorithm may refer to the existing technology and will notbe repeated here.

Further, after the optical flow vectors of the target feature points areobtained, based on the optical flow vectors of the target featurepoints, the depth information of the target feature points, and thevisual information of the target feature points, the moving object isrecognized. The visual information includes color and/or brightness.

FIG. 3 is a flowchart of a method for recognizing a moving objectaccording to an example embodiment of the present disclosure. As shownin FIG. 3, the method for recognizing the moving object includes thefollowing.

At 1021, a clustering process is performed on the obtained optical flowvectors of the target feature points to obtain at least one optical flowvector group, where a direction displacement between any optical flowvectors in the same optical flow vector group is smaller than a firstpre-set threshold (also referred to as a “direction threshold”), and anamplitude displacement between any optical flow vectors in the sameoptical flow vector group is smaller than a second pre-set threshold(also referred to as an “amplitude threshold”).

FIG. 4A is a schematic diagram illustrating optical flow vectors oftarget feature points according to an example embodiment of the presentdisclosure. The optical flow vectors in FIG. 4A can be expressed usingLucas-Kanade algorithm as follows:

${\begin{bmatrix}V_{x} \\V_{y}\end{bmatrix} = {\begin{bmatrix}{\sum_{i}{I_{x}\left( q_{i} \right)}^{2}} & {\sum_{i}{{I_{x}\left( q_{i} \right)}{I_{y}\left( q_{i} \right)}}} \\{\sum_{i}{{I_{y}\left( q_{i} \right)}{I_{x}\left( q_{i} \right)}}} & {\sum_{i}{I_{y}\left( q_{i} \right)}^{2}}\end{bmatrix}^{- 1}\begin{bmatrix}{- {\sum_{i}{{I_{x}\left( q_{i} \right)}{I_{t}\left( q_{i} \right)}}}} \\{- {\sum_{i}{{I_{y}\left( q_{i} \right)}{I_{t}\left( q_{i} \right)}}}}\end{bmatrix}}},$

where q_(i) is a point in a field of point P of the depth image. Thesize of the field of point P may be configured according to actualneeds, and is not limited by the present disclosure. For example, whenthe field of point P is a 5×5 field, the field of point P includes 25points. The optical flow vector corresponding to q_(i) is [V_(x)V_(y)]^(T), I_(x) and I_(y) are gradient information of point q_(i) ofthe depth image in x and y directions, respectively, and I_(t) is abrightness change of q_(i) between the two image frames.

In one embodiment, based on the above optical flow vectorrepresentation, a clustering algorithm (e.g., K-means++ algorithm) inunsupervised machine learning is used to perform the clustering processon the optical flow vectors of the target feature points. As such, theposition [u v]^(T) of each of the target feature points in the depthimage, the color and/or brightness of the target feature points, and theoptical flow vectors [V_(x) V_(y)]^(T) are used in the clusteringprocess to obtain at least one optical flow vector group in FIG. 4B. Theclustered optical flow vector group can be expressed as follows:

[uvV _(x) V _(y)(u,v,t)]^(T).

The above expression is merely illustrative and not a limitation of thepresent disclosure.

At 1022, based on depth information and visual information of each ofthe target feature points, the moving object is recognized from the atleast one optical flow vector group.

In some embodiments, based on the depth information of each of theobtained target feature points and the color and/or brightness of eachof the obtained feature points, a floodfill algorithm is used torecognize the moving object from at least one obtained optical flowvector group.

At 103, a moving speed vector of the moving object is determined.

In one embodiment, based on the three-dimensional coordinates of themoving object in the world coordinate system in the pre-set number ofimage frames, the moving speed vector of the moving object isdetermined.

At 104, based on the moving speed vector of the moving object, a regionwhere the moving object may collide with the movable platform (alsoreferred to as a “potential collision region”) is determined, such thatthe movable platform is controlled to perform the obstacle avoidanceprocess in the potential collision region.

In some embodiments, based on the moving speed vector of the movingobject and the moving speed vector of the movable platform, thefollowing equations are used to determine a moving path of the movableplatform, and based on the moving path of the movable platform, theregion where the moving object may collide with the movable platform isdetermined:

${\begin{bmatrix}x \\y \\z\end{bmatrix} = {K\left\lbrack {R_{cw}\left\{ \ {\begin{bmatrix}x_{w} \\y_{w} \\z_{w} \\1\end{bmatrix} + {{V_{object}^{w} \cdot \Delta}\; t} + T_{wc} + {{V_{drone}^{w} \cdot \Delta}\; t}} \right\}} \right\rbrack}},{\begin{bmatrix}u \\v \\1\end{bmatrix} = {{z\begin{bmatrix}{x/z} \\{y/z} \\{z/z}\end{bmatrix}}.}}$

In some embodiments, after the moving path of the movable platform isobtained, the moving path of the movable platform is projected onto thedepth image. Based on the moving path of the movable platform, theregion where the moving object may collide with the movable platform isdetermined in the depth image. Based on the coordinate information ofvarious regions of the depth image in the world coordinate system, thethree-dimensional coordinates, in the world coordinate system, of theregion where the moving object may collide with the movable platform aredetermined. As such, the movable platform is controlled to perform thepre-set obstacle avoidance process at the three-dimensional coordinates.For example, after the collision region is determined, the movableplatform is controlled to move in a direction opposite to a currentmoving direction, or the moving path of the movable platform is adjustedto circumvent the potential collision region, or the movable platform iscontrolled to stop moving for a pre-set time period, thereby achievingthe objective of obstacle avoidance. The embodiments are illustrativeand do not limit the obstacle avoidance method of the presentdisclosure. In practical applications, the obstacle avoidance process isconfigured to satisfy actual needs.

In some embodiments, to increase the interaction between users andimprove user experience, the moving path of the movable platform may bedisplayed, and the potential collision region may be displayed to notifythe user to timely perform the obstacle avoidance process.

The present disclosure provides the method and the device for obstacleavoidance and the movable platform. The optical flow vectors of thetarget feature points in the depth image photographed by thephotographing device mounted at the movable platform and the movingspeed vector of the movable platform are obtained. Based on the obtainedoptical flow vectors of the target feature points, and the depthinformation and the visual information of the target feature points, themoving object is recognized from the depth image. Through tracking thethree-dimensional coordinates of the moving object in the worldcoordinate system in the pre-set number of the image frames, the movingspeed vector of the moving object is determined. As such, based on themoving speed vector of the moving object and the moving speed vector ofthe movable platform, the region where the moving object may collidewith the movable platform is determined, and the movable platform iscontrolled to perform the obstacle avoidance process in the potentialcollision region.

In the embodiments of the present disclosure, the moving object isrecognized and based on the moving speed vector of the moving object,the region where the moving object may collide with the movable platformis determined, such that the movable platform is controlled to avoid themoving object, and the safety and the user experience of the movableplatform are improved. The obstacle avoidance method can in particulareffectively prevents the moving object from affecting the movableplatform when the movable platform is the car moving on the ground orother movable object or a UAV hovering above the ground. In addition,because the intermediate result of the visual odometry (VO) algorithm ofthe existing movable platform generates the optical flow vectors of thetarget feature points required by the obstacle avoidance method, theintermediate result of the existing VO algorithm may be directlyobtained to detect the moving object. Thus, the amount of calculation iseffectively reduced, the efficiency of the obstacle avoidance detectionis increased, and the real-time performance of the obstacle avoidancedetection is further improved.

The present disclosure also provides an obstacle avoidance device. FIG.5 is a structural block diagram of an obstacle avoidance deviceaccording to an example embodiment of the present disclosure. As shownin FIG. 5, the obstacle avoidance device 10 is disposed at a movableplatform 20. The obstacle voidance device 10 includes a processor 11 anda photographing device 21. The processor 11 is communicatively coupledwith the photographing device 21. The photographing device 21 isconfigured to photograph a depth image. The processor 11 is configuredto obtain a depth image photographed by the photographing device 21,recognize a moving object based on the depth image, determine a movingspeed vector of the moving object, determine a region where the movingobject may collide with the movable platform 20 based on the movingspeed vector of the moving object, and control the movable platform 20to perform the obstacle avoidance process in the potential collisionregion.

In some embodiments, the processor 11 is configured to obtain theoptical flow vectors of the target feature points in the depth image.The target feature points do not include feature points of stationaryobjects. The moving object is recognized based on the optical flowvectors of the target feature points.

In some embodiments, the processor 11 is configured to obtain theoptical flow vectors of the target feature points in the depth imagebased on the visual odometry (VO) algorithm.

In some embodiments, the processor 11 is configured to extract thefeature points from the depth image photographed by the photographingdevice 21 mounted at the movable platform 20 based on the pre-set cornerpoint detection algorithm, determine the optical flow vectors of thefeature points through tracking relative positions of the feature pointsof two images frames, and remove the optical flow vectors of the featurepoints of the stationary objects from the optical flow vectors of thefeature points based on the three-dimensional coordinates of the featurepoints in the world coordinate system in the two image frames, to obtainthe optical flow vectors of the target feature points.

In some embodiments, the processor 11 is configured to, based on therelative positions of the feature point in the two image frames,determine a first displacement of a relative position of a feature pointon a succeeding image frame relative to the relative position of thefeature point in a preceding image frame and a second displacement ofthe relative position of the feature point in the preceding image framerelative to the relative position of the feature point in the succeedingimage frame, and determine the optical flow vectors based on the firstdisplacement or the second displacement if a relationship between thefirst displacement and the second displacement satisfies a pre-set firstpriori condition.

In some embodiments, the processor 11 is configured to determine anessential matrix corresponding to the three-dimensional coordinates ofthe feature points in the world coordinate system in the two imageframes based on a pre-set second priori condition, and remove theoptical flow vectors of the feature points of the stationary objectsfrom the optical flow vectors of the feature points based on theessential matrix and using the random sampling consensus algorithm, toobtain the optical flow vectors of the target feature points. The secondpriori condition is a condition relationship between thethree-dimensional coordinates of the feature points in the worldcoordinate system in the two image frames and the essential matrix.

In some embodiments, the processor 11 is configured to determineposition and attitude of the photographing device when photographing thetwo image frames based on the three-dimensional coordinates of thefeature points in the world coordinate system in the two image frames,and remove the optical flow vectors of the feature points of thestationary objects from the optical flow vectors of the feature pointsbased on the position and attitude of the photographing device whenphotographing the two image frames and using the random samplingconsensus algorithm, to obtain the optical flow vectors of the targetfeature points.

In some embodiments, the processor 11 is configured to recognize themoving object based on the optical flow vectors of the target featurepoints, depth information of the target feature points, and visualinformation of the target feature points, where the visual informationincludes color and/or brightness.

In some embodiments, the processor 11 is configured to perform aclustering process on the obtained optical flow vectors of the targetfeature points to obtain at least one optical flow vector group, andrecognize the moving object from the at least one optical flow vectorgroup based on the depth information and the visual information of eachof the target feature points. A direction displacement between anyoptical flow vectors in the same optical flow vector group is smallerthan a first pre-set threshold and an amplitude displacement between anyoptical flow vectors in the same optical flow vector group is smallerthan a second pre-set threshold.

In some embodiments, the processor 11 is configured to use a floodfillalgorithm to recognize the moving object from the at least one opticalflow vector group based on the depth information and the visualinformation of each of the target feature points.

In some embodiments, the processor 11 is configured to: determine amoving speed vector of the moving object based on the three-dimensionalcoordinates of the moving object in the world coordinate system in apre-set number of image frames.

In some embodiments, the processor 11 is configured to determine amoving path of the movable platform based on the moving speed vector ofthe moving object and the moving speed vector of the movable platform,and determine the region where the moving object may collide with themovable platform based on the moving path of the movable platform.

In some embodiments, the processor 11 is configured to project themoving path onto the depth image to determine the region where themoving object may collide with the movable platform in the depth image,and determine the three-dimensional coordinates of the potentialcollision region in the world coordinate system based on the depthimage.

In some embodiments, the processor 11 is configured to control themovable platform to move in a direction opposite to a current movingdirection of the movable platform in the potential collision region.

In some embodiments, the processor 11 is configured to adjust the movingpath of the movable platform, such that the movable platform circumventsthe potential collision region.

In some embodiments, the processor 11 is configured to control themovable platform to stop moving for a pre-set time period to avoid themoving object.

In some embodiments, the photographing device 21 includes at least onecamera.

In the embodiments of the present disclosure, the obstacle avoidancedevice performs the obstacle avoidance method. The operation principleand beneficial effects of the obstacle avoidance device are same as orsimilar to those of the embodiments of the obstacle avoidance method andwill not be repeated here.

The present disclosure also provides a movable platform. The movableplatform includes a body, a power system mounted at the body forsupplying power to the movable platform, and the disclosed obstacleavoidance device. The movable platform may be any of an unmanned aerialvehicle, an automobile, VR glasses, and AR glasses.

The present disclosure provides another obstacle avoidance device. FIG.6 is a structural block diagram of another obstacle avoidance deviceaccording to an example embodiment of the present disclosure. As shownin FIG. 6, the obstacle avoidance device 30 is disposed at a groundterminal 40. The obstacle avoidance device 30 includes a processor 31and a communication interface 32. The processor 31 is communicativelycoupled with the communication interface 32. The communication interface32 is configured to obtain a depth image photographed by a photographingdevice 51 mounted at a movable platform 50. The processor 31 isconfigured to recognize a moving object based on the depth image,determine a moving speed vector of the moving object, determine a regionwhere the moving object may collide with the movable platform based onthe moving speed vector of the moving object, and control the movableplatform to perform the obstacle avoidance process in the potentialcollision region.

In some embodiments, the processor 31 is configured to obtain theoptical flow vectors of the target feature points in the depth image.The target feature points do not include feature points of stationaryobjects. The moving object is recognized based on the optical flowvectors of the target feature points.

In some embodiments, the processor 31 is configured to obtain theoptical flow vectors of the target feature points in the depth imagebased on the visual odometry (VO) algorithm.

In some embodiments, the processor 31 is configured to extract thefeature points from the depth image photographed by the photographingdevice mounted at the movable platform based on the pre-set corner pointdetection algorithm, determine the optical flow vectors of the featurepoints through tracking relative positions of the feature points of twoimages frames, and remove the optical flow vectors of the feature pointsof the stationary objects from the optical flow vectors of the featurepoints based on the three-dimensional coordinates of the feature pointsin the world coordinate system in the two image frames, to obtain theoptical flow vectors of the target feature points.

In some embodiments, the processor 31 is configured to: based on therelative positions of the feature point in the two image frames,determine a first displacement of a relative position of a feature pointin a succeeding image frame relative to the relative position of thefeature point in a preceding image frame and a second displacement ofthe relative position of the feature point in the preceding image framerelative to the relative position of the feature point in the succeedingimage frame, and determine the optical flow vectors based on the firstdisplacement or the second displacement if a relationship between thefirst displacement and the second displacement satisfies a pre-set firstpriori condition.

In some embodiments, the processor 31 is configured to: determine anessential matrix corresponding to the three-dimensional coordinates ofthe feature points in the world coordinate system in the two imageframes based on a pre-set second priori condition, and remove theoptical flow vectors of the feature points on the stationary objectsfrom the optical flow vectors of the feature points based on theessential matrix and using the random sampling consensus algorithm, toobtain the optical flow vectors of the target feature points. The secondpriori condition is a condition relationship between thethree-dimensional coordinates of the feature points in the worldcoordinate system in the two image frames and the essential matrix.

In some embodiments, the processor 31 is configured to: determineposition and attitude of the photographing device when photographing thetwo image frames based on the three-dimensional coordinates of thefeature points in the world coordinate system in the two image frames,and remove the optical flow vectors of the feature points on thestationary objects from the optical flow vectors of the feature pointsbased on the position and attitude of the photographing device whenphotographing the two image frames and using the random samplingconsensus algorithm, to obtain the optical flow vectors of the targetfeature points.

In some embodiments, the processor 31 is configured to: recognize themoving object based on the optical flow vectors of the target featurepoints, depth information of the target feature points, and visualinformation of the target feature points, where the visual informationincludes color and/or brightness.

In some embodiments, the processor 31 is configured to: perform aclustering process on the obtained optical flow vectors of the targetfeature points to obtain at least one optical flow vector group, andrecognize the moving object from the at least one optical flow vectorgroup based on the depth information and the visual information of eachof the target feature points. A direction displacement between anyoptical flow vectors in the same optical flow vector group is smallerthan a first pre-set threshold and an amplitude displacement between anyoptical flow vectors in the same optical flow vector group is smallerthan a second pre-set threshold.

In some embodiments, the processor 31 is configured to: use a floodfillalgorithm to recognize the moving object from the at least one opticalflow vector group based on the depth information and the visualinformation of each of the target feature points.

In some embodiments, the processor 31 is configured to: determine amoving speed vector of the moving object based on the three-dimensionalcoordinates of the moving object in the world coordinate system in apre-set number of image frames.

In some embodiments, the processor 31 is configured to: determine amoving path of the movable platform based on the moving speed vector ofthe moving object and the moving speed vector of the movable platform,and determine the region where the moving object may collide with themovable platform based on the moving path of the movable platform.

In some embodiments, the processor 31 is configured to: project themoving path onto the depth image to determine the region where themoving object may collide with the movable platform in the depth image,and determine the three-dimensional coordinates of the potentialcollision region in the world coordinate system based on the depthimage.

In some embodiments, the obstacle avoidance device further includes adisplay assembly 33. The display assembly 33 is communicatively coupledwith the processor 31. The display assembly 33 is configured to displaythe moving path of the movable platform.

In some embodiments, the display assembly 33 is configured to displaythe depth image and mark the potential collision region in the depthimage.

In some embodiments, the processor 31 is configured to: control themovable platform to move in a direction opposite to a current movingdirection of the movable platform in the potential collision region.

In some embodiments, the processor 31 is configured to: adjust themoving path of the movable platform, such that the movable platformcircumvents the potential collision region.

In some embodiments, the processor 31 is configured to: control themovable platform to stop moving for a pre-set time period to avoid themoving object.

In some embodiments, the photographing device includes at least onecamera.

In the embodiments of the present disclosure, the obstacle avoidancedevice performs the obstacle avoidance method. The operation principleand beneficial effects of the obstacle avoidance device are same as orsimilar to those of the embodiments of the obstacle avoidance method andwill not be repeated here.

In the embodiments of the present disclosure, the disclosed device andmethod may be implemented in other manners. For example, the describeddevice embodiments are intended to be illustrative. For example,division of units is merely a logic function division. In actualimplementations, the units may be divided differently. For example, aplurality of units or components may be combined or integrated intoanother system, and certain features may be omitted or not implemented.Further, the displayed or discussed mutual coupling or direct couplingor communicative coupling may be indirect coupling or communicativecoupling through certain interfaces, devices or units, and may beelectrical, mechanical, or in other form.

The units described as separate parts may or may not be physicallyseparated. The parts displayed as units may or may not be physicalunits, that is, may not be located in one place, and may be distributedin a plurality of network units. Some or all modules may be selectedaccording to actual requirements to achieve the objectives of thetechnical solution of the present disclosure.

In addition, the functional units in the embodiments of the presentdisclosure may be integrated into one processing module, may bedistributed to a plurality of physically separate units, or may have twoor more units integrated into one module. The integrated modules may beimplemented in hardware or in hardware plus software function modules.

The integrated unit implemented in software function units may be storedin a computer-readable storage medium. The software function unitsstored in the storage medium includes a plurality of instructions tocause a computer (a personal computer, a server, or a network device) ora processor to perform some or all processes described in embodiments ofthe present disclosure. The storage medium includes a USB disk, aportable hard drive, a read-only memory (ROM), a random-access memory(RAM), a magnetic disk, an optical disk, and any other medium that canstore program code.

Those skilled in the art may understand that for the convenience andbrevity of the description, the above described function module divisionis intended to be illustrative. In practical applications, the functionsmay be distributed to different function modules for implementation.That is, the internal structure of the device may be divided intodifferent function modules to achieve all or some function s describedabove. For the detail of the operation principle of the device,reference may be made to the corresponding process in the foregoingmethod embodiments, and details are not described herein again.

Various embodiments of the present disclosure are merely used toillustrate the technical solution of the present disclosure, but thescope of the present disclosure is not limited thereto. Although thepresent disclosure has been described in detail with reference to theforegoing embodiments, those skilled in the art should understand thatthe technical solution described in the foregoing embodiments can stillbe modified or some or all technical features can be equivalentlyreplaced. Without departing from the spirit and principles of thepresent disclosure, any modifications, equivalent substitutions, andimprovements, etc., shall fall within the scope of the presentdisclosure. The scope of the invention should be determined by theappended claims.

What is claimed is:
 1. An obstacle avoidance method comprising:obtaining a depth image photographed by a photographing device mountedat a movable platform; recognizing, based on the depth image, a movingobject; determining a moving speed vector of the moving object;determining, based on the moving speed vector of the moving object, apotential collision region where the moving object is likely to collidewith the movable platform; and controlling the movable platform toperform an obstacle avoidance process in the potential collision region.2. The method of claim 1, wherein recognizing the moving object based onthe depth image includes: obtaining optical flow vectors of targetfeature points in the depth image, the target feature points not beingon a stationary object; and recognizing the moving object based on theoptical flow vectors of the target feature points.
 3. The method ofclaim 2, wherein obtaining the optical flow vectors of the targetfeature points in the depth image includes obtaining the optical flowvectors of the target feature points based on a visual odometry (VO)algorithm.
 4. The method of claim 3, wherein obtaining the optical flowvectors of the target feature points based on the VO algorithm includes:extracting feature points from the depth image based on a pre-set cornerpoint detection algorithm; determining optical flow vectors of thefeature points by tracking relative positions of the feature points intwo image frames; and removing, based on three-dimensional coordinatesof the feature points in a world coordinate system in the two imageframes, the optical flow vectors of the feature points on the stationaryobject from the optical flow vectors of the feature points to obtain theoptical flow vectors of the target feature points.
 5. The method ofclaim 4, wherein determining the optical flow vectors of the featurepoints by tracking the relative positions of the feature points in thetwo image frames includes, for one feature point of the feature points:determining, based on the relative positions of the one feature point inthe two image frames: a first displacement of the relative position ofthe one feature point in a succeeding one of the two image framesrelative to the relative position of the one feature point in apreceding one of the two image frames, and a second displacement of therelative position of the one feature point in the preceding one of thetwo image frames relative to the relative position of the one featurepoint in the succeeding one of the two image frames; and determining theoptical flow vector of the one feature point based on the firstdisplacement or the second displacement in response to a relationshipbetween the first displacement and the second displacement satisfies apre-set priori condition.
 6. The method of claim 4, wherein removing theoptical flow vectors of the feature points on the stationary objects toobtain the optical flow vectors of the target feature points includes:determining an essential matrix corresponding to the three-dimensionalcoordinates of the feature points in the world coordinate system in thetwo image frames based on a pre-set priori condition, the prioricondition including a condition relationship between thethree-dimensional coordinates of the feature points in the worldcoordinate system in the two image frames and the essential matrix; andremoving the optical flow vectors of the feature points on thestationary objects from the optical flow vectors of the feature pointsbased on the essential matrix and using a random sampling consensusalgorithm, to obtain the optical flow vectors of the target featurepoints.
 7. The method of claim 4, wherein removing the optical flowvectors of the feature points on the stationary objects to obtain theoptical flow vectors of the target feature points includes: determiningposition and attitude of the photographing device at a time ofphotographing the two image frames based on the three-dimensionalcoordinates of the feature points in the world coordinate system in thetwo image frames; and removing the optical flow vectors of the featurepoints on the stationary objects from the optical flow vectors of thefeature points based on the position and attitude of the photographingdevice at the time of photographing the two image frames and using arandom sampling consensus algorithm, to obtain the optical flow vectorsof the target feature points.
 8. The method of claim 2, whereinrecognizing the moving object includes recognizing the moving objectbased on the optical flow vectors of the target feature points, depthinformation of the target feature points, and visual information of thetarget feature points, the visual information including at least one ofcolor or brightness.
 9. The method of claim 8, wherein recognizing themoving object includes: performing a clustering process on the opticalflow vectors of the target feature points to obtain an optical flowvector group, a direction displacement between any optical flow vectorsin the optical flow vector group being smaller than a directionthreshold, and an amplitude displacement between any optical flowvectors in the optical flow vector group being smaller than an amplitudethreshold; and recognizing the moving object from the optical flowvector group based on the depth information and the visual informationof each of the target feature points.
 10. The method of claim 9, whereinrecognizing the moving object from the optical flow vector groupincludes recognizing the moving object from the optical flow vectorgroup using a floodfill algorithm based on the depth information and thevisual information of each of the target feature points.
 11. The methodof claim 1, wherein determining the moving speed vector of the movingobject includes determining the moving speed vector of the moving objectbased on three-dimensional coordinates of the moving object in a worldcoordinate system in a pre-set number of image frames.
 12. The method ofclaim 1, wherein determining the potential collision region includes:determining a moving path of the movable platform based on the movingspeed vector of the moving object and a moving speed vector of themovable platform; and determining the potential collision region basedon the moving path of the movable platform.
 13. The method of claim 12,wherein determining the potential collision region includes: projectingthe moving path onto the depth image to determine the potentialcollision region in the depth image; and determining three-dimensionalcoordinates of the potential collision region in the world coordinatesystem based on the depth image.
 14. The method of claim 12, furthercomprising: displaying the moving path of the movable platform.
 15. Themethod of claim 13, further comprising: displaying the depth image; andmarking the potential collision region in the depth image.
 16. Themethod of claim 1, wherein controlling the movable platform to performthe obstacle avoidance process in the potential collision regionincludes controlling the movable platform to move in a directionopposite to a current moving direction of the movable platform in thepotential collision region.
 17. The method of claim 1, whereincontrolling the movable platform to perform the obstacle avoidanceprocess in the potential collision region includes adjusting a movingpath of the movable platform to circumvent the potential collisionregion.
 18. The method of claim 1, wherein controlling the movableplatform to perform the obstacle avoidance process in the potentialcollision region includes controlling the movable platform to stopmoving for a pre-set time period to avoid the moving object.
 19. Themethod of claim 1, wherein the photographing device includes at leastone camera.